-
Notifications
You must be signed in to change notification settings - Fork 505
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Per environment master shard #538
base: master
Are you sure you want to change the base?
Conversation
Running cucumber gives the error: cannot load such file -- test/unit (LoadError)
These weren't supposed to make it into this PR
This is failing with the stacktrace: Caused by: LoadError: cannot load such file -- arel/visitors/bind_visitor /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require' /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require' /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require' /usr/src/app/.bundle/ruby/2.5.0/gems/ibm_db-4.0.0/lib/active_record/connection_adapters/ibm_db_adapter.rb:13:in `<top (required)>' /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require' /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require' /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency' /usr/src/app/.bundle/ruby/2.5.0/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require' /usr/src/app/lib/octopus/proxy_config.rb:245:in `initialize_adapter'
The gem install does this, but the download is slow. Watching the wget progress is more satisfying.
Error message is: ActiveRecord::StatementInvalid: NoMethodError: undefined method `[]' for nil:NilClass: from octopus/lib/octopus/abstract_adapter.rb:23:in `octopus_shard' from octopus/lib/octopus/abstract_adapter.rb:12:in `instrument' from vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.11.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log' This has shown to surface in the ibm_db and sqlserver adapters.
Error message is: ActiveRecord::StatementInvalid: NoMethodError: undefined method `[]' for nil:NilClass: from octopus/lib/octopus/abstract_adapter.rb:23:in `octopus_shard' from octopus/lib/octopus/abstract_adapter.rb:12:in `instrument' from vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.11.1/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log' This has shown to surface in the ibm_db and sqlserver adapters.
This allows the InstrumenterDecorator to properly log the shard name
Running tests individually works, but running the whole migration_spec.rb file will recreate the issue _almost_ every time, in a random spec. Here's the full error and stack trace: Octopus::Migration should send the query to the correct shard Failure/Error: delegate :adapter_name, :add_transaction_record, :case_sensitive_modifier, :type_cast, :to_sql, :quote, :quote_column_name, :quote_table_name, :quote_table_name_for_assignment, :supports_migrations?, :table_alias_for, :table_exists?, :in_clause_length, :supports_ddl_transactions?, :sanitize_limit, :prefetch_primary_key?, :current_database, :combine_bind_parameters, :empty_insert_statement_value, :assume_migrated_upto_version, :schema_cache, :substitute_at, :internal_string_options_for_primary_key, :lookup_cast_type_from_column, :supports_advisory_locks?, :get_advisory_lock, :initialize_internal_metadata_table, :release_advisory_lock, :prepare_binds_for_database, :cacheable_query, :column_name_for_operation, :prepared_statements, :transaction_state, :create_table, to: :select_connection RuntimeError: An unexpected error occurred during retrieval of table metadata: uncaught throw :"Fetch Failure: <error message could not be retrieved>" # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1858:in `rescue in tables' # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1845:in `tables' # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:55:in `table_exists?' # ./lib/octopus/proxy.rb:25:in `table_exists?' # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/schema_migration.rb:20:in `block in table_exists?' # ./gemfiles/.bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/deprecation/reporting.rb:36:in `silence' # ./gemfiles/.bundle/ruby/2.5.0/gems/activesupport-5.0.7.2/lib/active_support/deprecation/instance_delegator.rb:20:in `silence' # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/schema_migration.rb:20:in `table_exists?' # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:48:in `create_table' # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/connection_adapters/abstract/schema_statements.rb:1008:in `initialize_schema_migrations_table' # ./lib/octopus/proxy.rb:200:in `initialize_schema_migrations_table' # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1127:in `initialize' # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1017:in `new' # ./gemfiles/.bundle/ruby/2.5.0/gems/activerecord-5.0.7.2/lib/active_record/migration.rb:1017:in `run' # ./lib/octopus/migration.rb:187:in `block in run_with_octopus' # ./lib/octopus/proxy.rb:86:in `block (2 levels) in run_queries_on_shard' # ./lib/octopus/proxy.rb:345:in `using_shard' # ./lib/octopus/proxy.rb:85:in `block in run_queries_on_shard' # ./lib/octopus/proxy.rb:329:in `keeping_connection_proxy' # ./lib/octopus/proxy.rb:84:in `run_queries_on_shard' # ./lib/octopus/proxy.rb:93:in `block in send_queries_to_multiple_shards' # ./lib/octopus/proxy.rb:92:in `map' # ./lib/octopus/proxy.rb:92:in `send_queries_to_multiple_shards' # ./lib/octopus/migration.rb:186:in `run_with_octopus' # ./spec/support/octopus_helper.rb:46:in `migrate_to_version' # ./spec/support/octopus_helper.rb:37:in `migrating_to_version' # ./spec/octopus/migration_spec.rb:86:in `block (2 levels) in <top (required)>' # ------------------ # --- Caused by: --- # UncaughtThrowError: # uncaught throw :"\xA3\xC9\x8E+H\xEA\xC9\vX\x82g\x8B\xAAU\x00\x00\bY\r\x8C\xAAU\x00\x00\x1A\xD0\xC1\xE2\xCF\xE9\x1Dd0\xB3m\x8B\xAAU\x00\x00\x80T\r\x8C\xAAU\x00\x00\x01\x99\xD1o," # /usr/src/ruby-ibmdb/IBM_DB_Adapter/ibm_db/lib/active_record/connection_adapters/ibm_db_adapter.rb:1845:in `fetch_assoc'
This keeps the test suite going for most users that won't be using the ibm_db gem
Docker dev system
Check that @config exists. Fixes thiagopradi#413
DB2 ibm_db gem support
ibmdb/ruby-ibmdb#89 Until that issue is officially resolved by IBM, I'm using my local fork for testing Octopus.
This is an oddball bug, deep in the DB2 C library. Disconnecting from the database during the Passenger fork process causes it to hang indefinitely
Since we don't disconnect from DB2, this spec fails.
In many setups, the default master shard might be a different database server in each environment. A single `master_shard` config setting doesn't fit. This patch first checks the shards.yml for `octopus` => Rails env => master_shard for a setting, and still falls back to `octopus` => `master_shard` if it doesn't exist
I cowboyed up a bit too fast on the first fix. This moves the per-environment master_shard selection to its own yaml key
@calh - any updates on this PR? is this ready for review / merge? |
@thiagopradi, yes, I'm using this in production right now as well, and it's a key requirement for how our environments are set up. We have different database server names for development and production, and the master shard setting needs to be there to properly initialize the rails app. I shouldn't have probably merged in the ibm_db branch along with this PR. The only commit that's needed here is 1d07220. If you wanted to merge this separately from the ibm_db PR I could close & recreate this with just that single commit. Thank you! |
In many setups, the default master shard might be a different database server in each environment. A single
master_shard
config setting doesn't fit. This patch first checks the shards.yml foroctopus
=>master_shards
=> Rails env for a setting, and still falls back tooctopus
=>master_shard
if it doesn't exist.Example: